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ABSTRACT 

Rapid proliferation of different types of interaction devices that use 
more natural channels (voice, touch, gesture) for interfacing with the digital medium 
illustrates the trend towards the creation of more "humane" interaction mechanisms. 
However, the current historical paradox is that modern technological advances are 
dramatically ahead of our understanding of their possible uses and meaning on a 
conceptual level. This paper presents an overview of some of the currently available 
interaction technologies, the conceptual barriers that limit their use and the case 
for the creation of interaction mechanisms that make abstract (virtual) information 
more tangible. Topics covered include: (1) direct manipulation, i.e., the use of a 

mouse (cursor) for pointing at and manipulating graphically represented objects; (2) 
problems with the traditional interface; (3) multimodal interaction, focusing on ways 
of making the output in human-computer interaction more tangible by using different 
sensory channels; (4) barriers and problems associated with building tangialities, the 
effects on interaction of conflicting, competing, redundant, and complementary 
information; and the case for building tangialities. (Contains 20 references.) (MES) 
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Abstract 



Rapid proliferation of different types of interaction devices that use 
more natural channels (voice, touch, gesture) for interfacing with the 
digital medium Illustrates the trend (and need) towards the creation of 
more 'humane' Interaction mechanisms. However, the current 
historical paradox is that modern technological advances are 
dramatically ahead of our understanding of their possible uses and 
meaning on a conceptual level. 

In this paper I will present an overview of some of the currently 
available interaction technologies, the conceptual barriers that limit 
their use and the case for the creation of interaction mechanisms that 
make abstract (virtual) information more tangible. 



Introduction 

The meanings associated with the adjective "tangible" in an on-line version 
of the Merriam-Webster dictionary (see references for URL) include: 

1 a : capable of being perceived especially by the sense of 
touch ; PALPABLE b ; substantially real ; MATERIAL 

2 : capable of being precisely identified or realized by the mind 
<her grief was tangible> 
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The listed synonym for "tangible" is PERCEPTIBLE, which in turn, has the 
following synonyms: SENSIBLE, PALPABLE, TANGIBLE, APPRECIABLE, 
PONDERABLE. The evolution of the term, starting with sense percepts 
related to the sense of touch and ending with precise mental identification 
and realization of abstract concepts (like 'grief in the definition above), 
corresponds, more or less, to my view on this topic. In this paper, I would like 
to make a case that association of virtual and abstract information with 
multimodal sensory experiences creates a new layer of knowledge and 
action spaces that is more natural and efficient for humans. These in- 
between domains, where interactions with virtual data produce tangible 
sensations, I dubbed tangialities (see Figure 1.). 
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Figure 1. Representation of different action/knowledge domains 



Please note that the way I define the term tangiality includes all sensory 
modalities and is not reduced just to those related to the sense of touch 
(haptic, cutaneous, tactile). 



Our body may be considered as the first interface between ourselves and the 
real world. The interactions were guided by our goals (intentions), carried out 
through actions, and repeated or corrected based on perception of the 
consequences of actions (observations). 

Body actions were soon enhanced through the use of tools (artifacts). 
Norman ( Norman 1991) introduced the concept of a cognitive artifact, as a 
tool that enhances cognitive operations. Although the enhancement of body 
actions is sometimes achieved by sheer magnification (using a lever, or 
inclined plane), the enhancement of cognitive operations is most often the 
consequence of changing the nature of the task. An example of a cognitive 
artifact that enhances our ability to memorize and recall events is a personal 
calendar. Instead of trying to rehearse and memorize all of the events for 
weeks to come, we have to remember only to write them down into the 
calendar, and to consult the calendar every day. In the context of tangialities, 
cognitive enhancements are also the consequence of changing the nature of 
the task. Most often a change is in shift from relying on formal, abstract 
operations as a means of gaining knowledge, to direct manipulation of data 
(properties) with instantaneously observable results. For example, in order to 
answer the problem illustrated in Figure 2., “are the dimensions of the 
smaller cubes exactly one-half of the larger one?” we may use conventional 
knowledge of algebra and solve the problem. Direct manipulation approach 
would be just to juxtapose two smaller cubes next to the larger one and the 
answer becomes self-evident. Note that the formal solution can be made 
harder by choosing different dimensions (for example, the height of the big 
cube could be 8.372914) but this does not influence the direct manipulation 
solution. 



I t 4 , 

8 / 7 ^ 8 

J 4 ( 4*2 = 8)= I 4 

Figure 2. Getting the resuit using abstract operation or direct 

manipuiation 

I would like to add another word of clarification. For the purposes of this 
paper, I am not going to address a very fruitful area of research often 
referred to as "tangible interfaces” (for example, see Ullmer, B., Ishii, H. 
2000, and Patten, J., Ishii, H., Hines, J., Pangaro, G. 2001). The cornerstone 
of this approach is in using real objects with desired physical (manipulable) 

/ properties as data representations/containers. These objects embody 

computation regardless of whether they are connected to a computer or not. 
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Although the areas of development of tangible interfaces and tangialities 
overlap, and will probably merge in the future, in this paper I will focus on 
procedures that endow data representations with tangible properties and 
thus make manipulations carried out on data available to our senses. 

Direct Manipulation 

The first human-computer interfaces were abstract, efficient and accessible 
only to expert users. They involved learning the vocabulary and syntax of a 
command language which was then used to initiate some operations on the 
digitally stored data, and often one needed to issue a separate command to 
see the results of the previous one. There was no continuity of interaction - 
once the command was issued, there was no way of interfering with the 
process (short of aborting it). There was also no sensory feedback that 
would provide relevant information about the operation on an experiential 
level. 

One of the first examples of a tangiality domain was the introduction of 
Graphical User Interface (GUI) and the concept of direct manipulation. 

"Direct manipulation", a somewhat misleading term, was introduced by Ben 
Shneiderman in 1983 to describe what we take today to be an integral part of 
human-computer Interaction - the use of a mouse (cursor) for pointing at and 
manipulating graphically represented objects. The crucial characteristics of 
direct manipulation are: a) continuous visibility of the manipulated object; b) 
all the actions carried out on the objects are rapid. Incremental and 
reversible; and c) the consequences of actions are Immediately visible 
(Shneiderman, 1983, 1998). 

What makes direct manipulation a tangiality domain is the fact that it 
provided continuous sensory input (visual and kinesthetic feedback from 
hand-on-mouse positions) while acting on abstract parameters (like location 
coordinates, adjacency, parallelism) of digitally represented data. In spite of 
the fact that the output in direct manipulation depended on a single sense 
(vision), it truly revolutionized human-computer interaction. 



Figure 3. "Direct manipulation". Notice that in spite of the term the 
movements of the mouse are (indirectly) mapped to the screen 

coordinates. 

Suddenly, anyone who could see and make hand movements could use the 
computer. However, it is the very success of the direct manipulation 
paradigm that is now one of the obstacles to creating even more efficient 
interfaces. 

Problems with the Traditional Interface 

As Malcolm McCullough aptly put it in his book "Abstracting 
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Craft" (McCullough, 1996), one of the problems with the traditional "point- 
and-click" interface was the increased separation of the hand and the eye. In 
performing operations on digital data, the eye was given a mayor role of 
identifying, focusing, monitoring and interpreting, while the hand was 
reduced to performing simple repetitive gestures. The fact that the rising 
number of repetitive motion injuries is associated with individuals working 
with computers indicates that this analysis is not just a handy metaphor. 
Shifting the control to the eye did not bring any benefits either. Besides its 
physiological role in finding and interpreting visual clues, in the traditional 
GUI the eye is forced to play the role of the white cane of the blind for the 
hand. All of the cursor guidance and positioning, often demanding a single 
pixel precision (in graphic programs), is done under the guidance of the eye. 
This leads to overstraining of this sensory channel to the point that computer 
operators ‘forget’ to blink, ultimately developing the chronic dry conjunctivitis 
of the eyes. 

In our other daily activities we rarely depend on single-sense feedback. 

Take, for example, the simple act of putting a pencil on the table. Although 
the eyes are involved, their role is more general - seeing whether the 
surface of the table is within our reach, and if it is clear of other objects. The 
actual act of putting the pen on the surface is guided more by cutaneous and 
proprioceptive clues, and augmented by discrete but definite auditory cues. 
This multimodal and complementary feedback is the reason our daily actions 
do not cause over-straining of any particular sense. 

Introducing Multimodal Interaction 

Although the term multimodal interaction encompasses both input and 
output, I will focus more on ways of making the output in human-computer 
interaction more tangible by using different sensory channels. This should 
not be taken as an indication that multimodal input (for example, using both 
speech and gestures) is of lesser importance. It just introduces another level 
of complexity that goes beyond the topic of this paper. 

The value of multimodal output (feedback) was recognized in HCI design 
and is currently most widely-spread as coupling of actions to sounds. Sound 
production is a standard part of modern computer systems, and it is 
astonishing that the value of consistent sound feedback was not recognized 
earlier and integrated into interface design guidelines. By coupling actions to 
sounds I do not mean often exotic "sound schemes" featuring drum-rolls for 
window closings and alarm clock sounds for warning messages. An example 
of consistent auditory feedback is a discrete "click" associated with opening 
of a new window in Microsoft Internet Explorer. The sound is so discrete that 
many users don't consciously perceive it, yet immediately notice its absence 
when they switch to another browser. 

Another indicator of this trend is the introduction of a mouse (Logitech, iFeel 
™ mouse) that allows the user to "feel" different objects (for example, 
folders) and actions (dragging, scrolling) in the traditional GUI interactions. 
Haptic feedback is provided by a vibro-tactile unit in the mouse and can be 
finely tuned to fit individual preferences. Although the additional information 
initially seems trivial and meager (a series of vibration patterns), it becomes 
very quickly evident that it significantly increases the comfort of interactions 
(Figure 4.). With complementary information about cursor location or action, 
the user does not have to rely as much on an already over strained sense of 
vision. The difference in the experience can be compared to the difference 
between typing on a standard keyboard where every key-press is 
accompanied by a tactile, kinesthetic and auditory cue, and typing using 
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keyboards where the keys are outlined on a touch-sensitive surface and 
provide no specific feedback. Numbers seem to confirm the benefits to the 
user of haptic technologies - in the first year they were introduced Logitech 
sold a quarter of a million of iFeel™ mice (quoted from Immersion 
TouchSense™ Web site). 




Hz 



/v__jva. 




time 



Figure 4. Movement of the iFeel mouse (cursor) over the table cells is 
augmented by haptic output produced by vibro-tactlle unit in the 
mouse. Although the stimulus is fairly discreet it significantly reduces 
the load of the visual sense during table input or selections using pulh 

down menus. 



Manifest increase in the quality of interaction experience is making the 
devices that use haptic or force-feedback a standard in the area of computer 
games. A quote from Briggs and Srinivasan illustrates the use of haptic 
interfaces in PC game playing: 

Active haptic interfaces can improve a user’s sense of 
presence: Haptic interfaces with 2 or fewer actuated degrees of 
freedom are now mass-produced for playing PC videogames, 
making them relatively cheap (about US$100 at the time of this 
writing), reliable, and easy to program. Although the complexity 
of the cues they can display is limited, they are surprisingly 
effective communicators. For example, if the joystick is vibrated 
when a player crosses a bridge (to simulate driving over 
planks) it can provide a landmark for navigation, and signal the 
vehicle’s speed (vibration frequency) and weight (vibration 
amplitude). 

(Briggs and Srinivasan, 2001) 

An important area where multimodal sensory feedback plays a crucial role is 
the area of affective computing. The pioneer of research in affective 
computing, Rosalind Picard (Picard, 1997) suggests a number of possible 
applications where the affective state of a human user becomes accessible 
to a computer, or another remote user. One of the applications is 
TouchPhone, developed by Jocelyn Sheirer (described in Picard, 2000), 
where the pressure that a participant in a phone conversation applies to the 
headset is transmitted to other party’s computer screen as a color range - 
blue corresponding to slight pressure and red corresponding to the maximum 
pressure value. Inspired by TouchPhone I dared imagine a more 
sophisticated model where haptic information is transmitted back and forth 
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by holding the other party’s simulated ’’hand” while carrying on the 
conversation (Figure 5.). 



Besides making abstract data manipulations tangible and accessible to 
humans and providing a channel for affective communication, using 
complementary sensory feedback to illustrate complex physical interactions 
is becoming a method of choice, especially for getting feedback while 
operating complex machinery or vehicles. The paradox here is that the 
human operator is most often not directly exposed to the relevant physical 




Figure 5. By "holding hands" one can carry out both verbal and non- 
verbal parts of a conversation. It remains to be seen whether the actual 
emotional "value" to the users is high enough for this invention to be 
economically viable, (photo and hand simulation by S. Milekic, modeled 

by M. Lengauer) 



changes, and these are made available by translating numerical data into a 
sensory experience readily interpretable by a human. The value of adding 
additional simulated sensory information to the real world task is beautifully 
illustrated by an example provided by cognitive scientist David Kirsh: 

This odd situation which digital technology creates is nicely 
portrayed by the way modern airplanes rely on simulations of 
the feel of flying to improve the control of pilots. Apparently, jets 
fly faster if their center of mass is moved closer to the plane’s 
nose, thereby changing the relative position of the center of 
mass with respect to the center of lift. The trouble is that in 
moving the center of mass forward there is an increased 
chance that the plane will tip into a nose dive. To keep the 
plane flying on this knife edge the speed and sensitivity of 
adjustments Is so great that pilots can no longer use 
mechanical means to control their planes. To assure fast 
enough response such jets now rely on digital networks to relay 
a simulated feel to the pilots. When a pilot pulls up on his 
steering wheel the computers inside the plane simulate the 
resistance of the ailerons delivering to the pilot the haptic 
Information he or she needs to know what they are doing. 

Small computer adjustments augment and speed up these pilot 
reactions. To the pilot this force feedback is an integral part of 
the way he or she flies the plane. But, of course, there is no 
true resistance in the steering wheel. Pulling harder on the 
wheel is just a way of sending the number 7 to the wing 
actuators instead of the number 5. Computation is so 
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irremediably built into planes that pilots could be in simulators. 

(Kirsh, 2001) 

There is yet another area, academically not that well researched, that readily 
embraced (no pun intended) the prospect of multisensory interaction. This is 
the vast domain of on-line sex. As NBC reports, there exist already a number 
of (multisensory) products that belong to the new “cyberdildonics” area, as 
well as full “cyber sex suits” that allow a wide variety of tactile sensations to 
be experienced on strategic body parts (Brunker, MSN NBC online). 

Barriers 

While it may seem that all arguments are in favor of building tangialities, it is 
worth investigating the barriers and problems associated with this approach. 
As I see them, there are some closely related general problems that are 
listed here separately only for the sake of clarity. These are: 

• success of "drag-and-drop” and "point-and-click" interface; 

• failure to realize that changing to, or addition of, another interaction 
device calls for redesign of the GUI; 

• need to un-learn established (imposed) conventions. 

Although it may seem to be a paradox that an early solution would be 
blocking the introduction of more advanced ones, historically this is a 
common occurrence. The QWERTY keyboard arrangement became 
standard for typewriters, and continued to be used even for computer 
keyboards where any conceivable keyboard layout is equally accessible. In 
the same way, the very success of mouse/cursor point-and-click interface 
made it a de facto standard closely associated with the very concept of what 
it means to interact with a computer. This standardized notion of what an 
interface should look like affected adversely the introduction of any new 
interaction device. For example, using a touchscreen as an interaction 
device while preserving the traditional GUI creates huge problems for the 
user because of the discrepancy in scale of objects necessary for 
comfortable interaction. An icon with dimension of 8 x 8 pixels; for example, 
a window closing box, is perfectly acceptable for a single pixel active tip of 
the cursor, but is definitely inappropriate for finger interaction. In 
environments adequately scaled for finger interactions, touchscreens have 
been shown to be superior pointing devices (Sears, Shneiderman, 1991). 

The same holds true for other interaction devices - introduction of continuous 
speech recognition calls for adequate feedback that verbal information has 
been successfully transmitted. The use of haptic mice and joysticks 
introduces the texture as a GUI design element, etc. 

Another general problem that has to be taken into account when introducing 
new ways of interaction is the need for un-learning of adopted conventions. 
This can be a very slow process, and a transitional stage should be a part of 
the design of any new convention (and lack thereof is often the reason for 
their failure). 

More specific problems with the introduction of multimodal feedback come 
from our lack of knowledge of the complexities of multimodal interaction. Just 
adding another channel to human-computer interaction is not by default 
beneficial. A logical and commonsensical analysis tells us that additional 
information presented through another channel (like any other information) 
may fall into the following categories: 
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• conflicting 

• competing 

• redundant 

• complementary 

In the above list, only the last category has a beneficial effect on interaction. 
In the following paragraphs I will provide examples of different types of 
multimodal information. 

Conflicting 

Conflicting information is often the consequence of hardware/bandwidth 
limitations as is the case with lack of audio/video synchronization in 
streamed Web- videos. Sometimes it is a product of poor design; for 
example, when an animated character's mouth movements are inappropriate 
for actual utterances. Conflicting information is sometimes purposefully 
designed into an application. An example is some Web site designs which try 
to keep the user "glued” to the site in order to artificially boost ratings. 

Competing 

An example of competing multimodal information is a voice overlay that is 
not synchronized to the printed text one is trying to read. Animated graphics 
(Gifs) on Web sites are another example of information competing for the 
same sensory modality (visual) and claiming a part of cognitive resources. 

An example from everyday life is the effect carrying on a phone conversation 
has on driving ability. 

Redundant 

A definition of redundant multimodal information conveyed through another 
sensory channel, which does not increase the total amount of information 
about the interaction but is also not adversely affecting the interaction. 

Complementary 

Complementary multimodal information is information conveyed through 
another sensory channel that does increase the total amount of information 
received and has a beneficial effect on interaction. This effect can be 
manifest as an increase in efficacy of interaction, or decrease in number of 
errors. This is the only instance where the bandwidth of human-computer 
information channel is Increased by engagement of another channel. 

Case for Building Tangialities 

In conclusion, one can make the case for building tangialities for the 
following reasons: 

• widening bandwidth of human/computer communication channel; 

• adding affective dimension to interaction; 

• allowing grasping and manipulation of complex concepts without the 
need for explicit formalization; 

• reducing cognitive load by use of intuitive body (biological) 
knowledge; 

• reducing the strain on one sense (vision) - single sense fatigue; 

• possibility of adding another dimension to meta data ~ “how does it 
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feel” (being able to feel the texture of paintings and other, othenwise 
“untouchable” objects) 

Another significant use of tangible descriptions of data and results of data 
manipulations is in making the digital domain more accessible for 
populations with special needs. There are already some promising results in 
this area (Yu, Ramloll and Brewster, 2001; Gouzman, Karasin, Braunstein 
2000 ). 

Currently, we are lacking a satisfactory theory of multimodal interaction and 
most often arrive at usable results through a process of trial and error. It is 
evident that this theory has to come from interdisciplinary efforts bridging the 
disciplines as diverse as neurophysiology, tele-robotics and computer 
science. 
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